Diversely interlinked data networks provide access to a vast array of information. Documents, pictures or videos may be retrieved through a network from a computer within the same building, or from a computer on the other side of the world. The explosion of information made available by networking computers drives the need for ever faster and more efficient ways to manage data processing.
Networking relies on communication protocols for processing streams of information transferred between network-connected computers. The information is typically transmitted in one or more data sets or data packets defined in accordance with a data communication protocol. Transmission Control Protocol/Internet Protocol (TCP/IP) is an example of a suite of communication protocols used for internet applications. TCP is the protocol used to establish a connection between two networked computers so that streams of data may be exchanged. TCP also establishes a method for ensuring delivery of the data and ensuring that information packets are delivered in the correct order. Internet protocol (IP) specifies the format of data packets, also called datagrams, transferred between networked computers. IP also specifies the addressing scheme used to transfer a data packet from one computer to another.
Widely networked systems present a number of challenges. For example, systems of disparate technologies must be interlinked using gateways or interface devices that provide a compatible communication protocol. The security of a private user or private network connected to a wide area network, including the largest wide area network, the internet, may be protected by virus detection and/or data security or encryption systems. Additionally, the vast amount of information available through networking must be meaningfully accessible through search engines or other data organization techniques.
Data sets, such as IP datagrams, may be examined, filtered, and reformatted to provide a user system or network with virus protection, intrusion detection, private network interconnectivity and text searching functionality. An intrusion detection system, such as a system implemented by a firewall, examines packets to filter messages that do not meet specified security criteria. A firewall may use a packet filter to inspect each packet entering or leaving the network. A packet is accepted or rejected based on a set of user-defined rules. The inspection may be implemented as a table lookup comparing various IP packet header fields to values in the table until either a matching entry in the table is found, or until no match is found and a default rule is selected. The disposition of the packet, as determined by the rule, determines whether a packet is forwarded, dropped silently, or dropped with a notification to the source host.
Another example of data set examination is the network translation required to connect a private network to the wider network. A local area network (LAN) may be connected to the internet through a central gateway device and using a Network Address Translation (NAT) implementation. The NAT translates between internal data traffic within the LAN and external data traffic to and from the internet. The NAT is located where the LAN meets the internet and makes the necessary IP address translations to the data packets. NAT implementations modify outgoing and incoming network packets to enable the interconnection between the two, possibly overlapping, address realms of the LAN and the internet. To distinguish between the LAN and internet addresses, NAT implementations keep a table of address mappings. When a packet is outbound from the LAN, the NAT software modifies the packet header so that the source address is the same as the gateway address. On inbound packets, the NAT software maps the destination address of the incoming packet (the gateway address) to the original source address based upon a NAT table lookup.
A virtual private network (VPN) provides another example of an implementation of data set examination and filtering. A virtual private network may constructed using public wires to connect nodes. These systems use encryption and other security techniques to ensure that only authorized users can access the network and that the information transferred through the VPN is secure. A VPN system must check packets in several ways, including checking to determine if an outbound packet must be encrypted, or to determine if an inbound packet is encrypted, or should have been encrypted. Each of these inspections requires the VPN application to compare the packet header information to a set of rules.
Typically, of the above examples of data set examination and analysis are implemented by a process that includes comparing portions of an information data set to a lookup table representing a predefined set of rules. As communication speeds increase, this task demands an increasing amount of system resources.